Content creation in augmented reality environment

ABSTRACT

Systems and methods for creating content in augmented reality (AR) environment. The content is created by a combination of at least a handheld device and an AR device, and involves initiation and establishment of a session between the handheld device and the AR device, command and data generation at the handheld device, transmission of the command and data from the handheld device to the AR device, processing of the command and data at the AR device to build context from the command and/or data and to create content from the context, saving and sharing of the created content, retrieval of the created content, and optionally modification of the retrieved content.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119to Indian Patent Application No. 201911033920, filed on Aug. 22, 2019,in the India Patent Office, the disclosure of which is incorporated byreference herein in its entirety.

BACKGROUND 1. Field

The present disclosure relates generally to augmented reality. Thedisclosure, more particularly, relates to content creation in augmentedreality environment.

2. Description of Related Art

Augmented reality is an enhanced version of reality where direct and/orindirect views of the physical real world environments are augmentedwith virtual superimposed computer-generated graphics, images,animations, 3D models, sounds and the like, to enhance a viewer'sperception of reality. In other words, augmented reality involves usingthe existing environment and overlaying information on it to make a newartificial environment. Through augmented reality information about thephysical real world can be made interactive to highlight certainobjects, enhance understandings, and provide data regarding the same.

Augmented reality can be displayed on a wide variety of displays, suchas screens of mobile devices or tablets, head-mounted displays,augmented realty glasses, and the like. Augmented reality can be usedfor purposes as simple as a text-messaging or as complicated as aninstruction on how to perform a surgical procedure.

There have been several endeavors to implement augmented reality basedapplications that allow creation and sharing of messages having textualcontent, notes, graphical content, animated content, and the like,superimposed on a real world environment. For example, augmented realitybased messaging applications such as ‘SNAPPY’, ‘TRACES’, ‘WALLAME’,‘JUST A LINEF’, etc. allow creation and sharing of such type ofmessages. However, these applications allow content/message creationonly on a mobile phone screen.

Similarly, augmented reality based devices such as MANOMOTION, HOLOLENS,LEAP MOTION, etc. are also available that support hand gestures andcarry out different functions based on the hand gestures. U.S. Pat. No.8,638,989 relates to LEAP MOTION and mentions methods and systems foridentifying shapes and capturing motions of human hand inthree-dimensional space. United States Patent publication US20120113223relates to the HOLOLENS device of Microsoft Corporation, and mentionstechniques for user-interaction in augmented reality wherein a user'stouch or hand gestures directly manipulate a user interface (i.e. thegraphics in the augmented reality). However, both LEAP MOTION andHOLOLENS are limited to hand gesture recognition and do not supportcreation and sharing of content/messages. Moreover, in such devicesthere is lack of active communication between a gesture generator whichis typically worn on a hand and an augmented reality display device,which may cause delays in carrying out the functions. Furthermore, thegeneration of gestures by hand movements is a cumbersome activity as itcan cause physical fatigue consequently leading to inaccurate gesturesand incorrect functions.

United States Patent publication US20110164000 relates to acommunicating stylus and United States Patent publication US2018018830relates to a smart pen with flexible display, both of which are capableof writing on any surface or in three-dimensional space. The writtenimages or text are saved and subsequently displayed on a display of acomputing device which introduces delay in the displaying. However, boththe communicating stylus and the smart pen are incapable of facilitatingcreation and sharing of content/messages based on augmented reality.

There is therefore felt a need of solution which enables augmentedreality based content creation and sharing, and at the same time alsoenables manipulation of the content in three-dimensional space.

SUMMARY

This summary is provided to introduce concepts of the disclosure relatedto content creation in AR environment, as disclosed herein. This summaryis neither intended to identify essential features of the disclosure asper the present disclosure nor is it intended for use in determining orlimiting the scope of the disclosure as per the present disclosure.

In accordance with an embodiment of the present disclosure, there isprovided a method for creating content in an augmented realityenvironment. The method comprises pairing a handheld device with anaugmented reality (AR) device. The method further comprises generating,by the handheld device, at least one command in connection with anobject of interest. The method further comprises generating, by thehandheld device, data in connection with an object of interest. Themethod further comprises transmitting, by the handheld device, the atleast one command and the data to the AR device. The method furthercomprises determining, by the AR device, a context in relation to theobject of interest corresponding to the at least one command and thedata. The method further comprises creating, by the AR device, a contentbased on the context and the data. The method further comprisesrendering, by the AR device, the created content.

The method further comprises authenticating, by the AR device or anexternal device, a content recipient. The method further comprisesauthenticating, by the AR device or the external device, at least onecondition set by a content creator for delivery of a content created bythe creator. The method further comprises notifying, by the AR device orthe external device to the content recipient, creation of the content bythe content creator upon successful authentication of the at least onecondition. The method further comprises retrieving, by the AR device,the created content. The method further comprises rendering, by the ARdevice, for the content recipient, the created content.

In accordance with another embodiment of the present disclosure, thereis provided a method for creating and retrieving content in an augmentedreality environment. The method comprises pairing a handheld device withan augmented reality (AR) device. The method further comprisesgenerating, by the handheld device, in response to an operation of aninput means of the handheld device performed by a content creator, afirst command in connection with an object of interest of the contentcreator. The method further comprises generating, by the handhelddevice, in response to a maneuvering of the handheld device by thecontent creator, data in connection with the object of interest of thecontent creator. The method further comprises transmitting, by thehandheld device, the first command and data to the AR device. The methodfurther comprises determining, by the AR device, a context in relationto the object of interest corresponding to the first command and data.The method further comprises creating, by the AR device, a content basedon the context and the data. The method further comprises rendering, bythe AR device, for the content creator, the created content. The methodfurther comprises authenticating, by the AR device or an externaldevice, a content recipient. The method further comprisesauthenticating, by the AR device or the external device, at least onecondition set by the content creator for delivery of the content createdby the creator. The method further comprises notifying, by the AR deviceor the external device to the content recipient, creation of the contentby the content creator upon successful authentication of the at leastone condition. The method further comprises generating, by the handhelddevice, in response to an operation of an input means of the handhelddevice performed by the content recipient, a second command inconnection with the created content. The method further comprisestransmitting, by the handheld device, the second command to the ARdevice. The method further comprises retrieving, by the AR device, thecreated content based on the second command. The method furthercomprises rendering, by the AR device, for the second user, the createdcontent.

In accordance with another embodiment of the present disclosure, thereis provided a system for content creation in an augmented realityenvironment. The system comprises a handheld device configured togenerate at least one command in connection with an object of interest.The handheld device is further configured to generate data in connectionwith the object of interest. The handheld device is further configuredto transmit the at least one command and the data. The system furthercomprises an augmented reality (AR) device, wherein the AR device ispaired with the handheld device. The AR device is configured to receivethe at least one command and the data. The AR device is furtherconfigured to determine a context in relation to the object of interestcorresponding to the at least one command and the data. The AR device isfurther configured to create a content based on the context and thedata. The AR device is further configured to render the created contenton a display unit of the AR device.

In accordance with another embodiment of the present disclosure, thereis provided a system for content creation in an augmented realityenvironment comprising a handheld device comprising a command generatorconfigured to generate at least one command in connection with an objectof interest. The handheld device further comprises a data generatorconfigured to generate data in connection with the object of interest.The handheld device further comprises a handheld device interface unitconfigured to transmit the at least one command and the data. The systemfurther comprises an augmented reality (AR) device, wherein the ARdevice is paired with the handheld device. The AR device comprises an ARdevice interface unit configured to receive the at least one command andthe data. The AR device interface unit includes a command and datadecoder configured to decode the at least one command and the data. TheAR device further comprises a context builder module configured todetermine a context in relation to the object of interest correspondingto the at least one command and the data. The AR device furthercomprises a content creator module configured to create a content basedon the context and the data. The AR device further comprises a displayunit configured to display the created content.

In accordance with another embodiment of the present disclosure, thereis provided a system for content creation in an augmented realityenvironment comprising a handheld device comprising an input means toenable generation of at least one command by an operation thereof to beperformed by a content creator. The handheld device further comprises aplurality of sensors configured to generate sensor data in response to amovement of the handheld device by the content creator. The handhelddevice further comprises a processor cooperating with the input means,the sensors and a memory. The processor is configured to execute a setof instructions stored in the memory to implement a command generatorconfigured to generate, in response to the operation of the input meansperformed by the content creator, the at least one command in connectionwith an object of interest. The processor is further configured toimplement a data generator configured to combine the sensor data togenerate object data in connection with the object of interest. Theprocessor is further configured to implement a handheld device interfaceunit configured to transmit the at least one command and the objectdata. The system further comprises an augmented reality (AR) device,wherein the AR device is paired with the handheld device. The AR devicecomprises a processor cooperating with a memory and configured toexecute a set of instructions stored in the memory. The processor isconfigured to implement an AR device interface unit configured toreceive the at least one command and the object data. The AR deviceinterface unit includes a command and data decoder configured to decodethe at least one command and the object data. The processor is furtherconfigured to implement a context builder module configured to determinea context in relation to the object of interest corresponding to the atleast one command and the object data. The processor is furtherconfigured to implement a content creator module configured to create acontent based on the context and the object data. The AR device furthercomprises a display unit configured to display the created content.

Before undertaking the DETAILED DESCRIPTION below, it may beadvantageous to set forth definitions of certain words and phrases usedthroughout this patent document: the terms “include” and “comprise,” aswell as derivatives thereof, mean inclusion without limitation; the term“or,” is inclusive, meaning and/or; the phrases “associated with” and“associated therewith,” as well as derivatives thereof, may mean toinclude, be included within, interconnect with, contain, be containedwithin, connect to or with, couple to or with, be communicable with,cooperate with, interleave, juxtapose, be proximate to, be bound to orwith, have, have a property of, or the like; and the term “controller”means any device, system or part thereof that controls at least oneoperation, such a device may be implemented in hardware, firmware orsoftware, or some combination of at least two of the same. It should benoted that the functionality associated with any particular controllermay be centralized or distributed, whether locally or remotely.

Moreover, various functions described below can be implemented orsupported by one or more computer programs, each of which is formed fromcomputer readable program code and embodied in a computer readablemedium. The terms “application” and “program” refer to one or morecomputer programs, software components, sets of instructions,procedures, functions, objects, classes, instances, related data, or aportion thereof adapted for implementation in a suitable computerreadable program code. The phrase “computer readable program code”includes any type of computer code, including source code, object code,and executable code. The phrase “computer readable medium” includes anytype of medium capable of being accessed by a computer, such as readonly memory (ROM), random access memory (RAM), a hard disk drive, acompact disc (CD), a digital video disc (DVD), or any other type ofmemory. A “non-transitory” computer readable medium excludes wired,wireless, optical, or other communication links that transporttransitory electrical or other signals. A non-transitory computerreadable medium includes media where data can be permanently stored andmedia where data can be stored and later overwritten, such as arewritable optical disc or an erasable memory device.

Definitions for certain words and phrases are provided throughout thispatent document, those of ordinary skill in the art should understandthat in many, if not most instances, such definitions apply to prior, aswell as future uses of such defined words and phrases.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Thesame numbers are used throughout the drawings to reference like featuresand modules.

FIG. 1 illustrates an overview of a system for content creation in ARenvironment, in accordance with an embodiment with the presentdisclosure.

FIG. 2A illustrates a flowchart depicting the operations involved in amethod for creating content in AR environment, in accordance with anembodiment with the present disclosure.

FIG. 2B-2C illustrates a flowchart depicting the operations involved ina method for creating and retrieving content in AR environment, inaccordance with an embodiment with the present disclosure.

FIG. 3 illustrates a block diagram of a system for content creation inAR environment, in accordance with another embodiment of the presentdisclosure.

FIG. 4 illustrates a pictorial view of one or more objects recognizedand tracked by the system of FIG. 3 .

FIG. 5A illustrates a flow diagram for creating action list(s) forcontent creation in AR environment by at least the system illustrated inFIG. 3 .

FIG. 5B illustrates a flow diagram for determining a context for contentcreation in AR environment by at least the system illustrated in FIG. 3.

FIG. 5C illustrates a flow diagram for rendering a created content in ARenvironment by at least the system illustrated in FIG. 3 .

FIG. 6 illustrates a detailed structural block diagram of a handhelddevice of a system for content creation in AR environment, in accordancewith yet another embodiment of the present disclosure.

FIG. 7 illustrates a detailed structural block diagram of an AR deviceof the system for content creation in AR environment as illustrated inFIG. 6 .

FIG. 8A illustrates a block diagram for gesture/movement recognitionemployed by the system illustrated in FIGS. 1, 3 and 6-7 , and FIG. 8Billustrates a flow diagram for gesture recognition.

FIG. 9 illustrates techniques for handwriting feature extractionemployed by the system for content creation in AR environment asillustrated in FIGS. 1, 3 and 6-7 .

FIG. 10 illustrates techniques for handwriting segmentation employed bythe system for content creation in AR environment as illustrated inFIGS. 1, 3 and 6-7 .

FIG. 11 illustrates techniques for handwriting recognition employed bythe system as illustrated in FIGS. 1, 3 and 6-7 .

FIGS. 12A-12B, 13-15 and 16A-16B illustrate different implementations ofthe present disclosure for creating content in AR environment.

It should be appreciated by those skilled in the art that any blockdiagrams herein represent conceptual views of illustrative methodsembodying the principles of the present disclosure. Similarly, it willbe appreciated that any flow charts, flow diagrams, and the likerepresent various processes which may be substantially represented incomputer readable medium and so executed by a computer or processor,whether or not such computer or processor is explicitly shown.

DETAILED DESCRIPTION

FIGS. 1 through 16B, discussed below, and the various embodiments usedto describe the principles of the present disclosure in this patentdocument are by way of illustration only and should not be construed inany way to limit the scope of the disclosure. Those skilled in the artwill understand that the principles of the present disclosure may beimplemented in any suitably arranged system or device.

The various embodiments of the present disclosure describe aboutdifferent systems and methods for creating content in augmented reality(AR) environment.

In the following description, for purpose of explanation, specificdetails are set forth in order to provide an understanding of thepresent disclosure. It will be apparent, however, to one skilled in theart that the present disclosure may be practiced without these details.One skilled in the art will recognize that embodiments of the presentdisclosure, some of which are described below, may be incorporated intoa number of systems.

However, the methods and systems are not limited to the specificembodiments described herein. Further, structures and devices shown inthe figures are illustrative of embodiments of the present disclosureand are meant to avoid obscuring of the present disclosure.

It should be noted that the description merely illustrates theprinciples of the present disclosure. It will thus be appreciated thatthose skilled in the art will be able to devise various arrangementsthat, although not explicitly described herein, embody the principles ofthe present disclosure. Furthermore, all examples recited herein areprincipally intended expressly to be only for explanatory purposes tohelp the reader in understanding the principles of the disclosure andthe concepts contributed by the inventor to furthering the art and areto be construed as being without limitation to such specifically recitedexamples and conditions. Moreover, all statements herein recitingprinciples, aspects, and embodiments of the disclosure, as well asspecific examples thereof, are intended to encompass equivalentsthereof.

In accordance with an embodiment of the present disclosure as shown inFIG. 1 , a system (100) for content creation in an augmented realityenvironment comprises a handheld device (110) (e.g., first electronicdevice) configured to generate at least one command in connection withan object of interest. The handheld device (110) is further configuredto generate data in connection with the object of interest. The handhelddevice (110) is further configured to transmit the at least one commandand the data. The system (100) further comprises an augmented reality(AR) device (120) (e.g., second electronic device), wherein the ARdevice (120) is paired with the handheld device (110). The AR device(120) is configured to receive the at least one command and the data.The AR device (120) is further configured to determine a context inrelation to the object of interest corresponding to the at least onecommand and the data. The AR device (120) is further configured tocreate a content based on the context and the data. The AR device (120)is further configured to render the created content on a display unit ofthe AR device (120).

In an aspect, an active communication is established between thehandheld device (110) and the AR device (120) to pair the handhelddevice (110) with the AR device (120), whereby the handheld device (110)is configured to initiate a session and the AR device (120) isconfigured to recognize and track the object of interest and lock theobject of interest to establish the session.

In an aspect, the handheld device (110) is configured to generate acombination of a command and data for tagging the object of interest,and the AR device (120) is configured to lock the object of interestupon receiving the command and data for tagging the object of interest.

In an aspect, the handheld device (110) includes an input means toenable generation of the at least one command by an operation thereof,and a plurality of sensors to enable generation of the data.

In an aspect, the input means is selected from button, click, capacitivetouch switch, resistive touch switch and piezo touch switch, and thesensors are selected from a pressure sensor, a gyro, an accelerometer, amagnetic sensor, a proximity sensor and a grip sensor.

In an aspect, the handheld device (110) is configured to generate thedata in the event of manoeuvre of the handheld device (110) inthree-dimensional (3D) space relative to the object of interest.

In an aspect, the handheld device (110) is manoeuvred in a directionselected from: a direction which simulates writing of one or morecharacters in 3D space, a direction which simulates drawing of one ormore characters or figures in 3D space, a direction which simulatesflicking of the handheld device (110), and a direction which simulateswhich simulates tapping on one or more menu options in 3D space.

In an aspect, the characters are selected from alphabets, numbers andsymbols; and figures include geometrical figures.

In an aspect, to determine the context, the AR device (120) isconfigured to capture at least one parameter selected from a globalcontext, a local context, and one or more other objects in a field ofview of the AR device (120) when recognizing the object of interest. TheAR device (120) is further configured to identify a profile of a contentcreator. The AR device (120) is further configured to identify thecontext of the object of interest based on combination of the at leastone command, the data in connection with the object of interest, thecaptured parameter and the identified profile of the content creator.The AR device (120) is further configured to render an action list and amenu option for content creation based on the identified context. The ARdevice (120) is further configured to analyse the rendered action listand the menu option. The AR device (120) is further configured to buildthe context based on the analysed action list, menu and the identifiedcontext, thereby determining the context.

In an aspect, the AR device (120) is further configured to aggregate theidentified context, classify the aggregated context, and infer theclassified context.

In an aspect, the global context includes data belonging to the contentcreator stored on a cloud server, and the local context includes, dateand time of recognizing the object of interest, geolocation of theobject of interest and physical conditions of the object of interest.

In an aspect, to create the content, the AR device (120) is configuredto capture the built context. The AR device (120) is further configuredto generate a subsequent context-aware action list and a subsequentcontext-aware menu, based on the built context and the data. The ARdevice (120) is further configured to render the subsequentcontext-aware action-list and the subsequent context-aware menu on thedisplay unit of the AR device (120) for content creation.

FIG. 1 generally illustrates an overview of the system (100) for contentcreation in an AR environment according to the aforesaid embodiment. Afirst user, referred to as the content creator, who is interested increating and sharing content has to use both the handheld device (110)and the AR device (120) for the same. Initially the handheld device(110) is paired with the AR device (120) to establish a session betweenthe handheld device (110) and the AR device (120). In order to pair thehandheld device (110) with the AR device (120), an active communicationis established between the handheld device (110) and the AR device(120). The handheld device (110) then initiates the session whereuponthe AR device (120) worn by the content creator recognizes and tracksone or more real world objects in connection with which the contentcreator is interested in creating content. The handheld device (110)then generates a combination of a command and data for tagging at leastone object of interest of the content creator and transmits the same tothe AR device (120) through the active communication (e.g.,communication link) therebetween. The AR device (120), upon receivingthe tagging command and data, then locks the object of interest of thecontent creator to establish the session. The AR device (120) typicallycomprises a camera whereby the real world objects in a field of view ofthe camera get recognized and tracked and the object of interest of thecontent creator gets locked by the AR device (120).

Once the session is established, the handheld device (110) generates atleast one command or a first command for creating content in connectionwith the object of interest of the content creator, and furthergenerates data for creating content in connection with the object ofinterest of the content creator. The handheld device (110) thentransmits the said command and data for creating content to the ARdevice (120) through the active communication therebetween. The ARdevice (120), upon receiving the said command and data for creatingcontent, interprets the said command and data and determines a contextbased on the same. In other words, the AR device (120) determines thatcontext which relates to the object of interest and corresponds to saidcommand and data for creating content. Thereafter, the AR device (120)creates the content based on the context and renders the same on adisplay of the AR device (120) for the content creator. The AR device(120) builds a graphic user interface (UI) and renders the content inthe UI to enable the content creator to save and share the same.

The handheld device (110) comprises an input means to enable generationof the commands. The content creator can operate the input means in aparticular manner to generate the commands. The input means can compriseone or more buttons, one or more clicks, one or more touch switches suchas, but not limited to, capacitive touch switch, resistive touch switch,piezo touch switch, and the like or a combination of all the aforesaidinput means. In case the input means comprises buttons, the contentcreator can operate the buttons in particular manner to generatecommands, wherein each manner of pressing the buttons constitutes anoperation and hence a command. For example, pressing a button apredetermined number of times can constitute a tagging/objectrecognition operation and a tagging/object recognition command fortagging the object of interest, pressing the button for a predeterminedshort-time duration can constitute an initiation operation and aninitiation command for initiating a session, operating the button bypressing the button for a predetermined long-time duration canconstitute a detection operation and a detection command to detect theshape of the object of interest, etc. In case the input means comprisestouch switches, the content creator can operate the touch switches inparticular manner to generate commands, wherein each manner of operatingthe touch switches can constitute an operation and hence a command. Forexample, increased capacitance/resistance level in acapacitive/resistive touch switch can constitute an initiation operationand an initiation command for initiating a session. In case the inputmeans comprises a combination of the buttons, switches etc., the contentcreator can operate the buttons and switches in particular manner togenerate commands.

The handheld device (110) also comprises a plurality of sensors toenable generation of data. The sensors include, but are not limited to,pressure sensor, gyro, accelerometer, magnetic sensor, proximity sensor,grip sensor, etc. The handheld device (110) can bemaneuvered/moved/stroked by the content creator to generate data. Themaneuvering/moving/stroking of the handheld device (110) can be carriedout in three-dimensional (3D) space relative to the object of interestto generate data. For example, the handheld device (110) can bemaneuvered in a direction which simulates writing of one or morecharacters in 3D space relative to the object of interest, drawing ofone or more characters in 3D space relative to the object of interest,flicking of the handheld device (110), tapping on the displayed UI, andthe like. The character can be alphabets, numbers, symbols, etc. Thefigures can be geometrical figures, freestyle shapes, and the like. Thesensors included in the handheld device (110) detect the maneuvering ofthe handheld device (110) and generate data based on the same. Forexample, the sensors, upon detecting the maneuvering/moving of thehandheld device (110) in a direction which stimulates writing of one ormore characters in 3D space relative to the object of interest, willgenerate data representing the characters. Further, for example, thesensors upon detecting the maneuvering/moving of the handheld device(110) in a direction which simulates tagging on an objection recognitionmenu option in the displayed UI, will generate data representing anobject recognition maneuver of the handheld device (110).

In order to interpret the said command and data for creating content asreceived from the handheld device (110) and determine the context, theAR device (120) performs additional operations. Firstly, the AR device(120) captures several parameters related to the object of interest andto the content creator. One of the parameters can be global contextrelated to the content creator. The global context comprises databelonging to the content creator available on the internet or stored oncloud storage systems, and can include, for example, data aboutlikes/dislikes of the content creator, activities of the contentcreator, etc. Another parameter can be local context related to theobject of interest, and can include, for example, data and time when theobject of interest is recognized and locked by the AR device (120),geolocation of the object of interest available typically from a GPSsystem incorporated in the AR device (120), and physical conditions ofthe object of interest. Another parameter can be the type of other realworld objects in the vicinity of the object of interest of the contentcreator, which other real world objects can be recognized by the ARdevice (120) in its field of view while recognizing the object ofinterest of the content creator. After capturing one or more of theaforesaid parameters, the AR device (120) then identifies a profile ofthe content creator available on the internet or stored on cloud storagesystems. The profile of the content creator can comprise informationrelated to the name, age, education, job, etc. of the content creator.

The AR device (120) then identifies the context of the object ofinterest from the command and data received from the handheld device(110), one or more of the captured parameters and the identified profileof the content creator. Based on the identified context, the AR device(120) then renders an action list and a menu option for content creationin the UI on the display of the AR device (120). The AR device (120)aggregates the identified context, classifies the identified contextinto various categories, and infers the classified context to render theaction list and the menu options on the display. The identified contextcan be classified into categories such as education related context,tourism related context, food related context, etc.; wherein educationrelated context can be inferred, for example, as a type of educationalsubject such as mathematics, science, geography, etc.; tourism relatedcontext can be inferred as say adventure tourism, religious tourism,etc.; food related context can be inferred as say Indian food,continental food, etc. to render the action list and the menu options onthe display. The AR device (120) then analyzes the rendered action listand the depth of the menu options and builds the context from theanalyzed action list, menu options and the identified context, therebydetermining the context.

In order to create the content, the AR device (120) captures the builtcontext, and renders a subsequent or next-level context-aware actionlist in the UI and a subsequent or next-level context-aware menu optionsin the UI, based on the built context, the current action list and menuoptions, and the generated data received from the handheld device (110).The subsequent/next-level action list is referred to as context-awareaction list as it makes available advanced actions in the UI which canbe performed based on the determined context. Similarly, thesubsequent/next-level menu options are referred to as context-aware menuoptions as they provide advanced options in the UI which are based onthe determined context. Each time a new command and/or data is receivedfrom the handheld device (110), the AR device (120) updates the UIaccordingly with next-level actions lists and menu options. The contentcreated from the actions lists and the menu options can be saved andshared by the content creator.

The system (100) for content creation as described above alsofacilitates retrieval of the created content. A second user, hereinafterreferred to as a content recipient, who is interested in reading theshared content has to use the AR device (120) and optionally thehandheld device (110) for retrieving the content. In order to retrievethe content, firstly, authentication of the content recipient is carriedout. The content recipient upon wearing the AR device (120) will beprompted to authenticate himself/herself, either through an externaldevice such as a mobile phone, computer, and the like and/or through theAR device (120) itself. Authentication through the external device maybe carried out through password based authentication, one time password(OTP) based authentication, challenge based authentication such asidentifying alphabetical characters, identifying objects in an image,and the like authentication techniques. Authentication through AR device(120) may be carried out by iris recognition, finger print scanning,voice recognition, and the like biometric authentication techniques.

After successful authentication of the content recipient, authenticationof one or more delivery conditions set by the content creator fordelivery of the created content is carried out. Delivery conditions cancomprise confirming the local context parameter such as geolocation ofthe object of interest, physical conditions of the object of interest,etc., confirming the presence of other real world objects in thevicinity of the object of interest, and the like. Authentication of thedelivery condition(s) can be carried out by the external device and/orthrough the AR device (120) itself. Upon successful authentication ofthe delivery condition(s), the content recipient is notified of thecreated content by the AR device (120) or the external device.Thereafter, the AR device (120) retrieves the created content andrenders the same on the display of the AR device (120) for the contentrecipient to read.

The retrieval of the created content can also involve use of thehandheld device (110). After content recipient is notified about thecreated content, the handheld device (110) is paired with the AR device(120) by establishing an active communication therebetween. The contentrecipient can then operate the input means of the handheld device (110)in a particular manner to generate a command and/or data for retrievingthe created content. The handheld device (110) then generates at leastone retrieval or a second command and/or data in connection with thecreated content. The handheld device (110) then transmits thesecond/retrieval command and/or data to the AR device (120) through theactive communication therebetween. The AR device (120), upon receivingthe command and data for creating content, interprets thesecond/retrieval command and/or data and retrieves the created contentand renders the same on the display of the AR device (120) for thecontent recipient to read.

The system (100) for content creation in an AR environment as describedherein above gives rise to a method (200 a) for creating content in anAR environment as depicted in the flowchart illustrated in FIG. 2A.Referring to FIG. 2A, the method comprises at operation 202—pairing ahandheld device (110) with an augmented reality (AR) device (120). Themethod further comprises at operation 204—generating, by the handhelddevice (110), at least one command in connection with an object ofinterest. The method further comprises at operation 206—generating, bythe handheld device (110), data in connection with the object ofinterest. The method further comprises at operation 208—transmitting, bythe handheld device (110), the at least one command and the data to theAR device (120). The method further comprises at operation210—determining, by the AR device (120), a context in relation to theobject of interest corresponding to the at least one command and thedata. The method further comprises at operation 212—creating, by the ARdevice (120), a content based on the context and the data. The methodfurther comprises at operation 214—rendering, by the AR device (120),the created content.

In an aspect of the method, the operation of pairing (202) the handhelddevice (110) with the AR device (120) includes establishing an activecommunication between the handheld device (110) and the AR device (120).The operation of pairing (202) further includes initiating, by thehandheld device (110), a session. The operation of pairing (202) furtherincludes recognizing, by the AR device (120), the object of interest.The operation of pairing (202) further includes locking, by the ARdevice (120), the object of interest to establish the session.

In an aspect of the method, the operation of locking the object ofinterest includes generating, by the handheld device (110), acombination of a command and data for tagging the object of interest;and locking, by the AR device (120), the object of interest uponreceiving the command and data for tagging the object of interest.

In an aspect of the method, the operation of generating the at least onecommand includes operating an input means of the handheld device (110).

In an aspect of the method, the operation of generating the dataincludes manoeuvring the handheld device (110) in three-dimensional (3D)space relative to the object of interest.

In an aspect of the method, the operation of manoeuvring the handhelddevice (110) is selected from: moving the handheld device (110) in adirection which simulates writing of one or more characters in 3D space,moving the handheld device (110) in a direction which simulates drawingof one or more characters or figures in 3D space, moving the handhelddevice (110) in a direction which simulates flicking of the handhelddevice (110), and moving the handled device (110) in a manner whichsimulates tapping on one or more menu options in 3D space.

In an aspect of the method, the operation of determining the contextincludes capturing at least one parameter selected from a globalcontext, a local context, and one or more other objects in a field ofview of the AR device (120) when recognizing the object of interest. Theoperation of determining the context further includes identifying aprofile of a content creator. The operation of determining the contextfurther includes identifying the context of the object of interest basedon combination of the at least one command, the data, the capturedparameter and the identified profile of the content creator. Theoperation of determining the context further includes rendering anaction list and a menu for creating the content based on the identifiedcontext. The operation of determining the context further includesanalysing the rendered action list and the menu. The operation ofdetermining the context further includes building the context based onthe analysed action list, menu and the identified context.

In an aspect of the method, the operation of identifying the contextincludes aggregating the identified context, classifying the aggregatedcontext and inferring the classified context.

In an aspect of the method, the operation of creating the contentincludes capturing the built context. The operation of creating thecontent further includes generating a subsequent context-aware actionlist and a subsequent context-aware menu, based on the built context andthe data. The operation of creating content further includes renderingthe subsequent action-list and the subsequent menu for creating thecontent.

The method further comprises authenticating, by the AR device (120) oran external device, a content recipient. The method further comprisesauthenticating, by the AR device (120) or the external device, at leastone condition set by the content creator for delivery of the contentcreated by the creator. The method further comprises notifying, by theAR device (120) or the external device to the content recipient,creation of the content by the content creator upon successfulauthentication of the at least one condition. The method furthercomprises retrieving, by the AR device (120), the created content. Themethod further comprises rendering, by the AR device (120), for thecontent recipient, the created content.

In an aspect of the method, the operation of retrieving the createdcontent includes pairing by the content recipient, the handheld device(110) with the AR device (120). The operation of retrieving the createdcontent further includes generating, by the handheld device (110), atleast one retrieval command in connection with the created content. Theoperation of retrieving the created content further includestransmitting, by the handheld device (110), the at least one retrievalcommand to the AR device (120).

The system (100) for content creation in an AR environment as describedherein above also gives rise to a method (200 b) for creating andretrieving content in an AR environment as depicted in the flowchartillustrated in FIG. 2B-2C. Referring to FIG. 2B-2C, the method comprisesat operation 216—pairing a handheld device (110) with an AR device(120). The method further comprises at operation 218—generating, by thehandheld device (110), in response to an operation of an input means ofthe handheld device (110) performed by a content creator, a firstcommand in connection with an object of interest of the content creator.The method further comprises at operation 220—generating, by thehandheld device (110), in response to a maneuvering of the handhelddevice (110) by the content creator, data in connection with the objectof interest of the content creator. The method further comprises atoperation 222—transmitting, by the handheld device (110), the firstcommand and data to the AR device (120). The method further comprises atoperation 224—determining, by the AR device (120), a context in relationto the object of interest corresponding to the first command and data.The method further comprises at operation 226—creating, by the AR device(120), a content based on the context and the data. The method furthercomprises at operation 228—rendering, by the AR device (120), for thecontent creator, the created content. The method further comprises atoperation 230—authenticating, by the AR device (120) or an externaldevice, a content recipient. The method further comprises at operation232—authenticating, by the AR device (120) or the external device, atleast one condition set by the content creator for delivery of thecontent created by the creator. The method further comprises atoperation 234—notifying, by the AR device (120) or the external deviceto the content recipient, creation of the content by the content creatorupon successful authentication of the at least one condition. The methodfurther comprises at operation 236—generating, by the handheld device(110), in response to an operation of an input means of the handhelddevice (110) performed by the content recipient, a second command inconnection with the created content. The method further comprises atoperation 238—transmitting, by the handheld device (110), the secondcommand to the AR device (120). The method further comprises atoperation 240—retrieving, by the AR device (120), the created contentbased on the second command. The method further comprises at operation242—rendering, by the AR device (120), for the second user, the createdcontent.

In an aspect of the method for creating and retrieving content, theoperation of pairing the handheld device (110) with the AR device (120)includes the operation of establishing an active communication betweenthe handheld device (110) and the AR device (120). This operation ofpairing further includes initiating, by the handheld device (110), asession by generating an initiation command in response to an initiatingoperation of the input means of the handheld device (110) performed bythe content creator. This operation of pairing further includesrecognizing, by the AR device (120), the object of interest. Thisoperation of pairing further includes locking, by the AR device (120),the object of interest upon receiving the combination of command anddata for tagging the object of interest, to establish the session.

In an aspect of the method for creating and retrieving content, theoperation of locking the object of interest includes generating, by thehandheld device (110), a combination of a command and data for taggingthe object of interest in response to an object recognition operation ofthe input means and an object recognition manoeuvre of the handhelddevice (110) performed by the content creator; and locking, by the ARdevice (120), the object of interest upon receiving the command and datafor tagging the object of interest.

In accordance with another embodiment of the present disclosure as shownin FIG. 3 , a system (300) for content creation in an augmented realityenvironment comprises a handheld device (310) comprising a commandgenerator (311) configured to generate at least one command inconnection with an object of interest. The handheld device (310) furthercomprises a data generator (312) configured to generate data inconnection with the object of interest. The handheld device (310)further comprises a handheld device interface unit (313) configured totransmit the at least one command and the data. The system (300) furthercomprises an augmented reality (AR) device (320), wherein the AR device(320) is paired with the handheld device (310). The AR device (320)comprises an AR device interface unit (321) configured to receive the atleast one command and the data. The AR device interface unit (321)includes a command and data decoder (321 a) configured to decode the atleast one command and the data. The AR device (320) further comprises acontext builder module (322) configured to determine a context inrelation to the object of interest corresponding to the at least onecommand and the data. The AR device (320) further comprises a contentcreator module (323) configured to create a content based on the contextand the data. The AR device (320) further comprises a display unit (330)configured to display the created content.

In an aspect, an active communication is established between thehandheld device (310) and the AR device (320) to pair the handhelddevice (310) with the AR device (320), whereupon the command generator(311) is configured to initiate a session and generate a command fortagging the object of interest, the data generator (312) is configuredto generate data for tagging the object of interest, and the handhelddevice interface unit (313) is configured to combine the command anddata for tagging the object of interest and transmit the combination ofthe command and data. Further, the AR device (320) further comprises anobject recognition and tracking module (324) configured to recognize theobject of interest and lock the object of interest upon receiving thecommand and data for tagging the object of interest through the activecommunication therebetween, to establish the session.

In an aspect, the handheld device (310) includes an input means (315) toenable generation of the at least one command by an operation thereof,and the command generator (311) is configured to generate the at leastone command based on the operation of the input means (315).

In an aspect, the input means (315) is selected from button (315 a),click (315 b), capacitive touch switch (315 c), resistive touch switchand piezo touch switch. For the sake of brevity, only few of the inputmeans are shown in FIG. 3 , and the input means is not intended to belimited to those shown in FIG. 3 .

In an aspect, the handheld device (310) includes a plurality of sensors(316) configured to generate data in the event of manoeuvre of thehandheld device (310) in three-dimensional (3D) space relative to theobject of interest, and the data generator (312) is configured tocombine data generated by the sensors (316) to generate the data inconnection with the object of interest.

In an aspect, the sensors are selected from a pressure sensor (316 a), agyro (316 b), an accelerometer (316 c), a magnetic sensor, a proximitysensor and a grip sensor. For the sake of brevity, only few of thesensors are shown in FIG. 3 , and the sensors are not intended to belimited to those shown in FIG. 3 .

Referring to FIGS. 5A and 5B, in an aspect, to determine the context,the context builder module (322) is configured to capture at least oneparameter selected from a global context (510), a local context (512),and one or more other objects (508) in a field of view of the AR device(320) when the object recognition and tracking module (324) recognizesthe object of interest. The context builder module (322) is furtherconfigured to identify a profile of a content creator. The contextbuilder module (322) is further configured to identify the context ofthe object of interest based on combination of the at least one command,the data (514) in connection with the object of interest, the capturedparameter and the identified profile of the content creator. The contextbuilder module (322) is further configured to render an action list(502) and a menu option (504) for content creation based on theidentified context. The context builder module (322) is furtherconfigured to analyse the rendered action list and the menu option. Thecontext builder module (322) is further configured to build the contextbased on the analysed action list, menu and the identified context.

In an aspect, the context builder module (322) is further configured toaggregate the identified context, classify the aggregated context, andinfer the classified context [refer FIG. 5B].

Referring to FIG. 5C, in an aspect, to create the content, the contentcreator module (323) includes a context-aware builder module (323 a)configured to capture the built context. The context-aware buildermodule (323 a) is further configured to generate a subsequentcontext-aware action list (502) and a subsequent context-aware menu(504), based on the built context (518) and the data (514). Thecontext-aware builder module (323 a) is further configured to render thesubsequent action-list and the subsequent menu on the display unit (330)of the AR device (320) for content creation (516).

In an aspect, the AR device (320) further comprises a plurality ofobject recognition and tracking sensors (325) connected to the objectrecognition and tracking module (324).

In an aspect, the object recognition and tracking sensors (325) areselected from camera (325 a), motion sensor (325 b), GPS (325 c) andcompass (325 d).

In an aspect, the AR device (320) includes a network communication unitand a plurality of context building sensors (326) connected to thecontext builder module (322). The context building sensors are selectedfrom audio sensor (326 a), light/illumination sensor (326 b) and eyetracker (326 c).

In an aspect, the handheld device (310) is selected from a stylus or apen, and the AR device (320) is selected from wearable AR glasses or acomputing device such mobile phone, tablet, laptop, and the like.

FIG. 3 illustrates a block diagram of the system (300) for contentcreation in AR environment in accordance with another embodiment of thepresent disclosure. The content creator who is interested in creatingand sharing content has to use both the handheld device (310), forexample, a stylus, a pen, and the like, and the AR device (320) forexample wearable AR glasses for the creating and sharing content.Initially the handheld device (310) is paired with the AR device (320)by establishing an active communication therebetween to establish asession between the handheld device (310) and the AR device (320). Theactive communication can be established through communication techniquessuch as BLUETOOTH, WIFI, and the like. The command generator (311) inthe handheld device (310) then initiates the session by generating acommand for tagging the object of interest, and the data generator (312)generates data for tagging the object of interest. The handheld deviceinterface unit (313) then combines the command and data for tagging theobject of interest and transmits the combination of the command and datato the AR device (320) through the active communication therebetween.The object recognition and tracking module (324) in the AR device (320)recognizes the object of interest through the camera (325 a) and uponreceiving the combination of the command and data for tagging the objectof interest, locks the object of interest to establish the session.

Once the session is established, the command generator (311) in thehandheld device (310) generates at least one command or a first commandfor creating content in connection with the object of interest of thecontent creator, and the data generator (312) generates data forcreating content in connection with the object of interest of thecontent creator. The handheld device interface unit (313) then transmitsthe said command and data for creating content to the AR device (320)through the active communication of BLUETOOTH, WIFI, and the likeestablished therebetween. The AR device interface unit (321) receivesthe said command and data for creating content, whereupon the commandand data decoder (321 a) decodes the said command and data and providesthe same to the context builder module (322). The context builder module(322) determines a context in relation to the object of interestcorresponding to the said command and data. In other words, the ARdevice (320) determines that context which relates to the object ofinterest and corresponds to said command and data for creating content.Thereafter, the content creator module (323) creates the content basedon the context and the data and renders the same on the display (330) ofthe AR device (320) for the content creator. The content creator module(323) builds a graphic user interface (UI) and renders the content inthe UI to enable the content creator to save and share the same.

In order to interpret the said command and data for creating content asreceived from the handheld device (310) and determine the context, thecontext builder module (322) in the AR device (320) performs additionaloperations. Firstly, the context builder module (322) captures severalparameters related to the object of interest and to the content creator.One of the parameters can be global context related to the contentcreator. The global context comprises data belonging to the contentcreator available on the internet or stored on cloud storage systems,and can include, for example, data about likes/dislikes of the contentcreator, activities of the content creator, etc. Another parameter canbe local context related to the object of interest, and can include, forexample, data and time when the object of interest is recognized andlocked by the object recognition and tracking module (324), geolocationof the object of interest available typically from the GPS (325 b)incorporated in the AR device (320), and physical conditions of theobject of interest. Another parameter can be the type of other realworld objects in the vicinity of the object of interest of the contentcreator, which other real world objects can be recognized by the objectrecognition and tracking module (324) in field of view of the camera(325 a) while recognizing the object of interest of the content creator.After capturing one or more of the aforesaid parameters, the contextbuilder module (322) then identifies a profile of the content creatoravailable on the internet or stored on cloud storage systems. Theprofile of the content creator can comprise information related to thename, age, education, job, etc. of the content creator.

The object recognition and tracking module (324) employs known imageprocessing techniques to recognize the real world objects. FIG. 4generally illustrates a pictorial view of the real world objectsrecognized and tracked by the object recognition and tracking module(324) through the camera (325 a).

Referring to FIGS. 5A and 5B, the context builder module (322) thenidentifies the context of the object of interest from the said commandand data received from the handheld device (310), one or more of thecaptured parameters and the identified profile of the content creator.Based on the identified context, the context builder module (322) thenrenders an action list (502) and a menu option (504) for contentcreation in the UI on the display (330) of the AR device (320). Thecontext builder module (322) aggregates the identified context,classifies the identified context into various categories, and infersthe classified context to render the action list and the menu options onthe display (330). The identified context can be classified intocategories such as education related context, tourism related context,food related context, etc.; wherein education related context can beinferred, for example, as a type of educational subject such asmathematics, science, geography, etc.; tourism related context can beinferred as say adventure tourism, religious tourism, etc.; food relatedcontext can be inferred as say Indian food, continental food, etc. torender the action list and the menu options on the display. The contextbuilder module (322) then analyzes the rendered action list and thedepth of the menu options and builds the context from the analyzedaction list, menu options and the identified context, therebydetermining the context.

Referring to FIG. 5C, in order to create the content, the context-awarebuilder module (323 a) in the content creator module (323) captures thebuilt context (518), and renders a subsequent or next-levelcontext-aware action list (506) in the UI and a subsequent or next-levelcontext-aware menu options (506) in the UI, based on the built context(518), the current action list (502) and menu options (504), and thegenerated data (514) received from the handheld device (310). Thesubsequent/next-level action list is referred to as context-aware actionlist as it makes available advanced actions in the UI which can beperformed based on the determined context. Similarly, thesubsequent/next-level menu options are referred to as context-aware menuoptions as they provide advanced options in the UI which are based onthe determined context. Each time a new command and/or data is receivedfrom the handheld device (310), the content creator module (323) updatesthe UI accordingly with next-level actions lists and menu options. Thecontent created from the actions lists and the menu options can be savedand shared by the content creator.

Referring now to FIG. 6 , a detailed structural block diagram of ahandheld device of a system (600) for content creation in AR environmentin accordance with yet another embodiment of the present disclosure, isillustrated. The handheld device (610) includes but is not limited to, astylus, a pen, and the like. The handheld device (610) comprises aninput means (615) including, but not limited to, button (615 a), clickswitch (615 b), capacitive touch switch (615 c), resistive touch switchand piezo touch switch to enable generation of at least one command byan operation thereof to be performed by a content creator. The handhelddevice (610) further comprises a plurality of sensors (616), includingbut not limited to, a pressure sensor (616 a), a gyro (616 b), anaccelerometer (616 c), a magnetic sensor/magnetometer (616 d), earthmagnetic field sensor (616 e) a proximity sensor (616 f), a grip sensor(616 g), configured to generate sensor data in response to a movement ofthe handheld device (610) by the content creator. The handheld device(610) further comprises a processor (611) cooperating with the inputmeans, the sensors and a memory (612). The processor (611) is configuredto execute a set of instructions stored in the memory (612) to implementa command generator configured to generate, in response to the operationof the input means performed by the content creator, the at least onecommand in connection with an object of interest. The processor (611) isfurther configured to implement a data generator configured to combinethe sensor data (514) to generate object data in connection with theobject of interest. The processor (611) is further configured toimplement a handheld device interface unit configured to transmit the atleast one command and the object data. The handheld device (610) furthercomprises a communication module (613) connected to the processor (611)to transmit the command and object data, through communication modulessuch as WIFI (613 a), BLUETOOTH (613 b), near-filed communication (NFC)(613 c), and the like.

The handheld device (610) further comprises a vibrator (614) toindicate, typically, the switching ON/OFF of the handheld device (610),a signal processing module (617) cooperating with the processor (611) toprocess, typically, the signals from the sensors and provide theprocessed signals back to the processor (611) to enable the processor togenerate data. The handheld device (610) further also comprise a battery(618) with a power management module (619) further connected to theprocessor (611) to supply power to the processor (611) as well as to theinput means, sensors, memory, communication module, vibrator and signalprocessing module of the handheld device (610).

The system (600) further comprises an augmented reality (AR) device(620), wherein the AR device (620) is paired with the handheld device(610). The AR device (620) includes, but is not limited to, wearable ARglasses, head-mounted device, and the like. The AR device (620)comprises a processor (621) cooperating with a memory and configured toexecute a set of instructions stored in the memory. The processor (621)is configured to implement an AR device interface unit configured toreceive the command and the object data. The AR device interface unitincludes a command and data decoder configured to decode the at leastone command and the object data. The processor (621) is furtherconfigured to implement a context builder module configured to determinea context in relation to the object of interest corresponding to the atleast one command and the object data. The processor (621) is furtherconfigured to implement a content creator module configured to create acontent based on the context and the object data.

The AR device (620) further comprises a display unit (622) configured todisplay the created content. The AR device (620) further comprises acamera (623) to capture one or more real world objects including theobject of interest of the content creator. The AR device (620) furthercomprises a vibrator (624) to indicate, typically, the switching ON/OFFof the AR device (620).

The AR device (620) further comprises a communication module (625)connected to the processor (621) to receive the command and object datatransmitted by the handheld device (610), through communication modulessuch as WIFI (625 a), BLUETOOTH (625 b), near-filed communication (NFC)(625 c), and the like. An active communication is established betweenthe handheld device (610) and the AR device (620) to pair the handhelddevice (610) with the AR device (620) and transmit/receive datatherebetween. The communication module (625) also comprises USB module(625 d) to transfer data between the AR device (620) and an externaldevice such as a computer, laptop, mobile, etc. The communication modulealso comprises a GPS module (625 e) to detect the geolocation of theobject of interest of the content creator.

The AR device (620) further comprises an eye tracking module (626) andfocus adjustment module (627) both cooperating with the processor (621)for visual enhancement of the display unit (622). The AR device (620)further comprises an audio module (628) cooperating with the processor(621) to aid the content creator and/or a content recipient.

The AR device (620) further comprises a plurality of sensors (629),including but not limited to, a biometric sensor (629 a), a gyro (629b), an accelerometer (629 c), a magnetic sensor (629 d), earth magneticfield sensor (629 e) a proximity sensor (629 f), a grip sensor (629 g),a gesture sensor (629 h) all cooperating with the processor (621) toprovide respective data to aid in content creation and/or contentretrieval.

The AR device (620) further comprises a battery (630) with a powermanagement module (631) further connected to the processor (621) tosupply power to the processor (621) as well as to the display, camera,vibrator, communication module, eye tracking module, focus adjustmentmodule, audio module, and sensors of the AR device (620).

Referring now to FIG. 8A, a block diagram is shown formovement/gesture/maneuver recognition employed by the handheld device(110, 310, 610). The sensors (316, 616) of the handheld devices (110,310, 610) are, typically, 9-axis MEMS sensor. For example, the gyro (316b, 616 b) is a tri-axial 16-bit gyroscope, the accelerometer (316 c, 616c) is tri-axial 16-bit accelerometer, the magnetometer (616 d) atri-axial 13-bit magnetometer. The processor cooperating with thesensors is configured to perform uncertainty reduction of the sensordata using sensor fusion techniques, like Kalman Filter and CNN forprecise and accurate tracking of the handheld device (610) in 3D space.The recognized motions are used for generation of data, which along withthe commands and current context, decides the next action.

Referring to FIG. 8B, a flow diagram for movement/gesture/maneuverrecognition is illustrated. The movement recognition can be divided inthree operations, viz. feature extraction which involves extraction ofuseful data from noise created as a result of the movement/maneuveringof the handheld device (110, 310, 610), spotting which involves findingsignificant handwriting data from sensors for continuous handheld devicemotion data, and recognition which involves recognizing complex,similarly shaped letters within a large number of classes and consistsof classification and segmentation.

Referring to FIG. 9 , the feature extraction technique employed by thehandheld device (110, 310, 610) as per the flow diagram of FIG. 8B, isshown. Inertial sensor measurements (802) commonly contain noise, sensordrift, cumulative errors, and the influence of gravitation error thatproduce inaccurate output. Preprocessing operations (804) such ascalibration and filters are used to eliminate noise and errors from theinertial signals. The accelerometer (316 c, 616 c) data consist of twocomponents' motion induced acceleration and gravity. The gravitycomponent is treated as noise and removed as it does not depend on theuser's hand motion. Feature extraction (806) provides values ofaccelerations (ax, ay, az), angular velocities (gx, gy, gz) and 3Dattitude of the device in quaternion (q0, q1, q2, q3) as featureparameters generated by hand movement for further processing andanalysis.

Referring to FIG. 10 , the spotting and handwriting segmentationtechnique employed by the handheld device (110, 310, 610) as per theflow diagram of FIG. 8B, is shown. Segmentation of continuous handmotion data (808) simplifies the process of handwriting classificationin free space. In real-time, the accelerometer (316 c, 616 c) andgyroscope (316 b, 616 b) data is processed to segment the continuoushand motion data into handwriting motion segments and non-handwritingsegments (810). The angular velocity and linear acceleration of a handmotion are two controlling parameters, they provide information todetermine the beginning and end boundaries of handwriting segments. Thehandheld device uses acceleration and temporal thresholds to determinehandwriting segmentation for spotting significant motion data, with highaccuracy (810).

Referring to FIG. 11 , the handwriting recognition technique employed bythe handheld device (110, 310, 610) as per the flow diagram of FIG. 8B,is shown. The handheld device uses Dynamic Time Warping (DTW) techniquewhich computes the distance between two gesture signals from the datareceived from the sensors (812). The quaternion output from the motionsensor is transformed into Euler sequences of rotation angles. Roll(ψ),pitch(θ), and yaw(φ), are used in addition to the accelerations (ax, ay,az) and angular velocities (gx, gy, gz), as feature parameters toefficiently track and classify handwriting in a meaningful and intuitiveway. In real-time, the DTW recognition technique computes the similaritybetween the sensor data and one or more templates. Sensor data isaccepted and classified to a class, which has the reduced warpingdistance and matched the threshold value of that class. The matchedvalue is then obtained from the template(s) and used for performingoperations in the AR device (120, 320, 620) for command and dataprocessing (814).

Referring now to FIGS. 12A-12B and 13-16 , different implementations ofthe present disclosure for creating content in AR environment, areillustrated. FIG. 12A illustrates an implementation of the presentdisclosure for dropping and editing contents in air at a touristlocation by using the handheld device and the AR device. As shown inFIG. 12A, content creation is initiated through the clicker of handhelddevice/stylus/pen. The handheld device/stylus/pen button press creates acallout menu which can be used to drop an air message. A user/contentcreator can use handheld device stroke data generated by the sensorsupon stroking/maneuvering of the handheld device to write message in thecallout. The handheld device clicker long press creates a contextualmenu for sharing and storing menu. FIG. 12B illustrates animplementation of the present disclosure for editing and sharing editedcontent by using the handheld device and the AR device. As shown in FIG.12B, the user/content creator can use the handheld device/stylus/penbutton press action with stroke movement to make a selection. Thehandheld device button long press action then creates a contextual menufor editing the selection. Thereafter, the handheld device clicker longpress action creates a contextual menu for sharing the selection,

FIG. 13 illustrates an implementation of the present disclosure forselecting, gathering and creating content from real world objects forsharing the content. As shown in FIG. 13 , a content creator initiatescontent creation with handheld device/stylus/pen click. The contentcreator then makes a selection using the handheld device/stylus/penbutton press action with stroke movement. The handheld device longbutton presses action results in processing the selected content,gathering additional information and generating contextual menu.Thereafter, the content creator selects an option from the menu andprovides handheld device stroke data for creating content along withpick and drop relevant information.

FIG. 14 illustrates an implementation of the present disclosure forcreating content from live stream video or images and retrieving thesame. As shown in FIG. 14 , the AR device worn by a content creatorintelligently detects the object in the purview of the content creator,and events the objects present inside the video. The content creator canthen select an object using the handheld device/stylus/pen button click.The content creator then creates content using handheld device data andactions for the target user and sets viewing conditions. Thereafter, thecontent creator can use handheld device long press action to generatecontextual menu. In this case a menu is created which lists shoppingoption for the item selected.

FIG. 15 illustrates an implementation of the present disclosure as asmart classroom with interactive and editable AR contents. As shown inFIG. 15 , a content creator initiates the content creation with handhelddevice/stylus/pen click. The handheld device stokes/movement/manoeuvringdata is processed as handwriting. The long press of handheld devicebutton generates contextual menu for text formatting where the contentcreator can change the font color to white. The content creator longpresses the clicker button to generate insert image/video. A videoselector menu is created where item can be browsed using handheld devicecapacitive touch sensor. When the content creator long presses thehandheld device button a contextual menu is generated for video control,to pause, play, forward video.

FIG. 16 illustrates an implementation of the present disclosure formultilevel book marking and AR notes creation and retrieval. As shown inFIG. 16 , a content creator can use the handheld device/stylus/penbutton press action with stroke movement to highlight the selection,whereupon a text formatting menu is generated to add additional styling.The content creator can use handheld device/stylus/pen button long pressaction with movement. The AR device detects the shape drawn by the userand a user note is added. The content creator can create content ofvarious level for display such as high-lighted text, notes etc. Thecontent creator can use the handheld device clicker press action to saveand share content. Sharing menu provides customization for the level ofsharing to each content recipient. For example, only highlighted contentis shared with content recipient #1, and full content is shared withcontent recipient #2.

At least some of the technical advantages offered the system and methodfor content creation in AR environment according to the presentdisclosure includes:

-   -   creating and sharing, context aware, customized, handwritten,        invisible and secure AR messages in 3D space;    -   creating smart and intelligent AR contents using multiple        contextual menus and action list; and    -   enabling the system to be used as a remote for camera,        presentation, and the like.

Although the present disclosure has been described with variousembodiments, various changes and modifications may be suggested to oneskilled in the art. It is intended that the present disclosure encompasssuch changes and modifications as fall within the scope of the appendedclaims.

What is claimed is:
 1. A method for creating content in an augmentedreality (AR) display device, the method comprising: establishing acommunication link with a handheld device; receiving a command and datarelated to an object from the handheld device via the communicationlink, wherein the command is a selection of the object among a pluralityof objects located a real world within a field of view of the AR displaydevice; recognizing the object located in the real world within thefield of view of the AR display device based on the received command anddata; determining a context related to the recognized object; creatingan AR content based on the determined context and the received data; andrendering the created AR content for display in an AR display of the ARdisplay device, wherein the command is generated based on a user inputobtained via the handheld device, wherein the data is generated based onmovement of the handheld device in three-dimensional (3D) space relativeto the object.
 2. The method of claim 1, further comprising: locking theobject to establish a session.
 3. The method of claim 2, wherein lockingthe object includes: locking the object upon receiving the command anddata related to the object from the handheld device.
 4. The method ofclaim 1, wherein determining the context includes: capturing at leastone parameter selected from a global context, a local context, and oneor more other objects located in the real world within the field of viewof the AR display device when recognizing the object; identifying aprofile of a content creator; identifying the context of the objectbased on a combination of the command, the data, the at least onecaptured parameter and the identified profile of the content creator;rendering an action list and a menu for creating the content based onthe identified context; analyzing the rendered action list and the menu;and building the context based on the analyzed rendered action list,menu and the identified context.
 5. The method of claim 4, wherein: theglobal context includes data belonging to the content creator stored ona server, and the local context is selected from one or more of date andtime of recognizing the object, geolocation of the object and physicalconditions of the object.
 6. The method of claim 4, wherein creating thecontent includes: generating a subsequent context-aware action list anda subsequent context-aware menu, based on the built context and thedata; and rendering the subsequent context-aware action-list and thesubsequent context-aware menu for creating the content.
 7. The method ofclaim 1, further comprising: authenticating a content recipient;authenticating at least one condition set by a content creator fordelivery of a content created by the content creator; notifying, by theAR display device to the content recipient, creation of the content bythe content creator upon successful authentication of the at least onecondition; retrieving the created content; and rendering, for thecontent recipient, the created content.
 8. The method of claim 7,wherein the at least one condition includes at least one of location ofthe object, physical conditions of the object, or confirming a presenceof other real world objects in a vicinity of the object.
 9. The methodof claim 1, wherein: the handheld device includes a stylus or a pen. 10.A AR display device comprising: a display; a communication module; and aprocessor configured to be operatively connected to the display and thecommunication module, wherein the processor is configured to: establisha communication link with a handheld device via the communicationmodule; receive a command and data related to an object from thehandheld device via the communication link, wherein the command is aselection of the object among a plurality of objects located in a realworld within a field of view of the AR display device; recognize theobject located in the real world within field of view of the AR displaydevice based on the received command and data; determine a contextrelated to the recognized object; create an AR content based on thedetermined context and the data; and render the created AR content fordisplay on the display, wherein the command is generated based on a userinput obtained via the handheld device, wherein the data is generatedbased on movement of the handheld device in three-dimensional (3D) spacerelative to the object.
 11. The AR display device of claim 10, whereinthe processor is further configured to: lock the object to establish asession.
 12. The AR display device of claim 11, wherein to lock theobject to establish the session, the processor is further configured to:lock the object upon receiving the command and data related to theobject from the handheld device.
 13. The AR display device of claim 10,wherein the processor is further configured to: capture at least oneparameter selected from a global context, a local context, and one ormore other objects located in the real world within the field of view ofthe AR display device when recognizing the object; identify a profile ofa content creator; identify the context of the object based on acombination of the command, the data, the at least one capturedparameter and the identified profile of the content creator; render anaction list and a menu for creating the content based on the identifiedcontext; analyze the rendered action list and the menu; and build thecontext based on the analyzed rendered action list, menu and theidentified context.
 14. The AR display device of claim 13, wherein: theglobal context includes data belonging to the content creator stored ona server, and the local context is selected from one or more of, dateand time of recognizing the object, geolocation of the object andphysical conditions of the object.
 15. The AR display device of claim13, wherein the processor is further configured to: generate asubsequent context-aware action list and a subsequent context-awaremenu, based on the built context and the data; and render the subsequentcontext-aware action-list and the subsequent context-aware menu forcreating the content.
 16. The AR display device of claim 10, wherein theprocessor is further configured to: authenticate a content recipient;authenticate at least one condition set by a content creator fordelivery of a content created by the content creator; notify, by the ARdisplay device to the content recipient, creation of the content by thecontent creator upon successful authentication of the at least onecondition; retrieve the created content; and render, for the contentrecipient, the created content.
 17. The AR display device of claim 16,wherein the at least one condition includes at least one of location ofthe object, physical conditions of the object, or confirming a presenceof other real world objects in a vicinity of the object.
 18. The ARdisplay device of claim 10, wherein: the handheld device includes astylus or a pen.